Testing configurations management

The current standard for managing configurations on various namespaces in the Energyworx platform, is the resource manager in the ewx-cli library, available either under the ewx-cli config command group or internally as ewx_cli.resources.Manager. An option that users have with the library is to request a specific set of configurations from a namespace and download all of them plus any dependencies them may have, to a directory. This directory can then be used later by the resource manager to upload all of the configurations to any other namespace, making sure to make appropriate modifications along the way.

Because of the versatility and familiarity of the resource manager, with the release of 24.01 , we added support for the resource manager to the pytest-bdd-ewx automated tests library used in Energyworx. Configurations directories that have been created by ewx-cli or later can be loaded into pytest-bdd-ewx with a new step definition as long as the directory exists within the repo. This article covers the new step definitions and how to go about using the resource manager's configurations directories in automated test suites.

Configurations directories

Configurations directories can be created with the resource manager using the ewx-cli config download command. The various options this command has can be viewed directly by executing the aforementioned command without providing any arguments, whereas documentation and user instructions on how this command works can be found in the README of the ewx-cli library.

Something that is added on top of the resource manager in pytest-bdd-ewx is support for “$-substitution”, like is already available for other configuration files in the testing library. This means that if for example the value of the datasource ID in a transformation configuration is written as TEST_DATASOURCE_${test_timestamp}, then this will be converted into TEST_DATASOURCE_2024_02_15_14_34_00 if the timestamp the testing session started was “2024-02-15T14:34:00”. This allows for creation of configurations whose values depend on the timestamp the tests started, thus allowing for reusing the same configuration over and over, but it will create a different datasource each time (in case of a transformation configuration). Variables available for $-substitution are all global variables in pytest_bdd_ewx/config.py, whose variable names are case-insensitive.

Once you have created a configurations directory with the resource manager, you will have to decide whether this should become the default resource manager or a named resource manager (see definitions further below). If the former, extra care has to be taken that the configurations directory you created contains ALL configurations required for all automated test suites that use the default resource manager. By default, the global configurations directory is located at <project_root>/resources/test-data/configs (its definition can be found in pytest_bdd_ewx/context.py as the tests_configs_path variable). When using named resource managers, the best location to put the configurations directory is probably in the same directory as where the features directory is located for your automated test suite.

Step definitions

Initializing resource manager

When you want to use a configurations directory through a resource manager, the first step that needs to be done is creating a resource manager based on that directory. There are two step definitions that are provided for this:

Given I create a resource manager from the global configurations directory

Given I create a resource manager with name "{manager_name}" from directory "{dirpath}"

The first creates a default resource manager from the directory stored at the path <project_root>/resources/test-data/configs. In other step definitions, this particular resource manager is referred to as the “default resource manager”. Using the default resource manager can be useful when one prefers to create a single resource manager for all automated tests combined, and simply always upload configurations from a single directory. This also allows one to keep all testing configurations in a single directory, in turn allowing configurations to be shared between different automated test suites (think of testing rules for example that can be useful in many different situations).

The second step definition instead creates a resource manager with a specific name and initializes it from a given path. The path can either be absolute (although this is probably not advisable) or relative to the <project_root>. In other step definitions, this resource manager is referred to as the “resource manager with name <manager_name>”. Using a named resource manager can be useful when one prefers to use separate configurations directories for separate automated test suites. This in turn allows for the configurations directory itself to be stored in the same location as where the automated test suite scenarios and features are defined, making it easier to find the configurations and identify which configurations are used where.

Each step can only be used once in an entire testing session. The default resource manager can obviously only be initialized once, whereas a named resource manager can only be initialized once for each unique name. Attempting to initialize a resource manager that already exists will generate an error; as will referencing a resource manager that has not been initialized yet.

Uploading configurations

After initializing a resource manager, configurations can be uploaded to the destination test namespace with several flavors of step definitions. In the following, each step variant has two different versions; one for the default resource manager and one for the named resource manager. Both step definitions will be shown together.

First of all, you can request from a resource manager to upload all configurations that it has:

Given I upload all configurations using the default resource manager

Given I upload all configurations using the resource manager with name "{manager_name}"

This simply uploads all configurations that are currently loaded in the requested resource manager to the destination test namespace. This step is often useful when using named resource managers for specific automated test suites, as you can use this step in the setup.feature file to setup the entire test suite in one go.

Besides uploading all configurations at once, each resource type supported by the resource manager has three sets of step definitions. Keep in mind that transform rules are not the same as rules in the resource manager, and decision trees are split into flow decision trees and rule decision trees.

Uploading a single configuration of a specific resource type with a specific name:

Given I upload the <resource_type> with name "{name}" using the default resource manager

Given I upload the <resource_type> with name "{name}" using the resource manager with name “{manager_name}”

Uploading all configurations of a specific resource type whose names match a regex pattern:

Given I upload all <resource_type_plural> matching the regex pattern "{pattern}" using the default resource manager

Given I upload all <resource_type_plural> matching the regex pattern "{pattern}" using the resource manager with name "{manager_name}"

Uploading all configurations of a specific resource type:

Given I upload all <resource_type_plural> using the default resource manager

Given I upload all <resource_type_plural> using the resource manager with name "{manager_name}"

Using any of these step definitions will upload all configurations of that resource type that matches what you requested PLUS all dependencies that these configurations have. So, for example, if you were to use the step Given I upload the market adapter with name "my_MA" ..., then this would also upload the transformation configuration this market adapter uses and remap the ID set in the market adapter configuration to be the ID of the created transformation configuration. Similarly, this transformation configuration probably depends on a datasource classifier and one or multiple channel classifiers. The channel mappings themselves probably depend on a few transform rules and might call an ingestion channel flow, which in turn depends on a few rules. This entire tree of configuration dependencies will be uploaded if they are directly or indirectly required by this single market adapter configuration. The resource manager also handles making sure that all IDs referenced within those configurations are properly remapped to what the IDs are on the namespace.

Setting up automated test suites using these specific step definitions is very useful when using the default resource manager for all tests, but are also useful in showing the intent of a specific test scenario. For example, if you are creating a test scenario that tests if the ingestion of a file is handled appropriately, the only configuration you truly care about is the market adapter with which that file needs to be ingested. Any configurations that market adapter itself needs to execute properly should not be the concern of the test itself. This also makes the tests more robust, because if you were to change the market adapter to no longer have a specific dependency, you can just leave the test scenario alone as it never referenced those dependencies.